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WIKIPEDIA SCALE: HIGH END 

Operating systems: Ubuntu, Windows, OS X 

■ 100s of watchers 

■ 1000s of edits 

■ 1000s of editors 

Popular websites: Google, Facebook 

■ 1000s of watchers 

■ 1000s of edits 

■ 1000s of editors 

Presidents: Barack Obama, George W. Bush 

■ 1000s of watchers 

■ 10000s of edits 

■ 1000s of editors 



WIKIPEDIA SCALE: LOW END 

• [[Lima Bean]] 

■ Fewer than 30 watchers 

■ 784 edits 

■ 487 editors 

• [[Augustan literature]] 

■ 40 watchers 

■ 487 edits 

■ 190 editors 

• [[Geography of Botswana]] 

■ Fewer than 30 watchers 

■ 156 edits 

■ 54 editors 

■ 14 years old 



OUR WIKI’S SCALE 

1. 47 active contributors 

2. 67272 edits 

3. 400k views in 3.5 years 



WATCHER DISTRIBUTION: DECEMBER 2014 



■ 1 watcher ■ 2 watchers ■ 3-5 watchers ■ 6+ watchers 




QUANTIFYING ACCOUNTABILITY 

1. Determine number of reviewers for each page 

2. Determine a way to rate contributors 

3. Combine (1) and (2) to identify weak points in our data 

4. Address weak points 



WHAT MEDIAWIKI ALREADY DOES 



WikipediA 

The Free Encyclopedia 


Main page 
Contents 
Featured content 
Current events 
Random article 
Donate to Wikipedia 
Wikipedia store 

Interaction 

Help 

About Wikipedia 
Community portal 
Recent changes 
Contact page 

Tools 


Article Talk 

Information for "NASA” 


Basic information 


Display title 

NASA 

Default sort key 

NASA 

Page length (in bytes) 

88,578 

Page ID 

18426568 

Page content language 

English (en) 

Page content model 

wikitext 

Indexing by robots 

Allowed 

Number of page watchers 

736 

Number of redirects to this page 

37 

Counted as a content page 

Yes 


What link? harp 





WHAT MEDIAWIKI ALREADY DOES 






WHAT MEDIAWIKI ALREAI 


Special page 


Unwatched pages 

Showing below up to 50 results starting with # 1 . 

View (previous 50 | next 50) (20 | 50 | 100 | 250 | 500) 

1 SharePoint (Watch) 

2. Confluence (Watch) 

3. Windows Vista (Watch) 

4 Lotus Notes (Watch) 





EXTENSIONiWATCH ANALYTICS 


Watch Analytics: Pages 


The state of the Wiki: 18863 watches of which 3.7% (700) are pending 



N ◄ ► 

First page Previous page Next page 


page 

# 

watches 

A# reviewed 
watches 

% 

pending 

Max 

pending 

time 

Average 
pending time 

SharePoint (Watch) 

1 

0 

100 

10 days 

10 days 

Confluence (Watch) 

2 

1 

50 

14 days 

14 days 

Google Reader (Watch) 

2 

1 

50 

14 days 

14 days 

Lotus Notes (Watch) 

3 

1 

67 

50 days 

31 days 






WATCHER o PAGE RELATIONSHIPS 






ACCOUNTABILITY EMERGENCY 

SHORT TERM GOALS 

1. Encourage users to do their reviews 

2. Eliminate singly-watched pages 



NONDESCRIPT WATCHLIST LINK 





CONFUSING WATCHLIST 


4 i 



Mediawiki 


Main page 
Get MediaWiki 
Get extensions 
Tech blog 
Contribute 

Support 
User help 
FAQ 

Technical manual 
Support desk 
Communication 

Development 
Bug tracker 
Code repository 
Code docs 


££> English & Jamesmontalvo3 Talk Preferences Beta Watchlist New messages (157) Contributions Logout 


Special page 

Watchlist 


Search 


Q. 


For Jamesmontalvo3 (View relevant changes | View and edit watchlist | Edit raw watchlist | Clear the watchlist) 


85 pages on your watchlist. not separately counting talk pages. Email notification is enabled. Pages that 
have been changed since you last visited them are shown in bold. 

Mark all pages visited 








CONFUSING WATCHLIST, PART 2 


Browse categories 
Community portal 
Recent changes 
Current issues 
Sandbox 

Tools 
Q Atom 
Upload file 
Special pages 
Printable version 
Project Status 

In other languages 


23 April 2015 

• (diff | hist) . . m VisualEditor; 13:56 . . (-14) . . Syum90 (talk | contribs) (Reverted edits by 
109.237.244.202 (talk) to last revision by Neil P. Quinn-WMF) 

22 April 2015 

• (Page translation log); 23:38 . . Shirayuki (talk | contribs) marked MediaWiki-Vagrant for translation 

• (diff | hist) . . MediaWiki-Vagrant; 13:17 . . (-38) . . Frimelle (talk | contribs) 

• (diff | hist) . . Talk:VisualEditor; 02:57 . . (+582) . . Alsee (talk | contribs) (^Success Criteria) 

• (Page translation log); 02:44 . . Shirayuki (talk | contribs) marked MediaWiki 1.21 for translation 

• (diff | hist) . . m MediaWiki 1.21; 02:43 . . (+1) . . Shirayuki (talk | contribs) (dummy edit to update 
translation pages) 

• (Page translation log); 02:43 . . Shirayuki (talk | contribs) marked MediaWiki 1.21 for translation 

21 April 2015 

• (diff | hist) . . Extension talk:Semantic Forms; 19:48 . . (+640) . . 109.158.179.155 (talk) (—> Getting 
CSS codes in dropdown list items when using a mapping template) 




PENDING REVIEWS 


A James Talk Admin links Preferences My pending reviews (3) Contributions 


Special page 

Pending reviews 


Search 


Q 


You have 3 pending reviews. 


Weekly Group Meeting - 2015/02/20 

1 Display 1 change since last visit 

* 

view page history 

• 9 Apr* User: Joe made an edit with summary: Added some important information 


International Space Station 

Display 2 changes since last visit 

view page history 

• 2 April UserJoe made an edit without a summary 

• 2 April User: Jane made an edit with summary: Reverted edits by Joe (talk) to last revision by Bob 

Heat Exchanger 

Display 3 changes since last visit 

4 

view page history 

h 4 


• 13 Apia UsenDoris approved a revision 

• T4 Apr* UserJoe made an edit without a summary 

• 14 Apri User: Joe made an edit with summary: Fixed installation procedure step 5 





HOW WELL ARE WE DOING WITH REVIEWERS? 






BIG IMPROVEMENTS 






AT THE BOTTOM OF PENDING REVIEWS 


A James Talk Admin links Preferences My pending reviews (3) Contributions 


Special page 

Pending reviews 


Search 


Q 


You have 3 pending reviews. 


Weekly Group Meeting - 2015/02/20 

1 Display 1 change since last visit 

* 

view page history 

• 9 Apri User: Joe made an edit with summary: Added some important information 


International Space Station 

Display 2 changes since last visit 

view page history 

• 2 April UserJoe made an edit without a summary 

• 2 April User: Jane made an edit with summary: Reverted edits by Joe (talk) to last revision by Bob 

Heat Exchanger 

Display 3 changes since last visit 

4 

view page history 

h 4 


• 13 Apia UsenDoris approved a revision 

• T4Apr« , User:Joe made an edit without a summary 

• t4>v>rtfUser:Joe made an edit with summary: Fixed installation procedure step 5 





WATCH SUGGESTIONS 


Heat Exchanger 


Display 3 changes since last visit I view page history 


• t3>v>rtUser:Doris approved a revision 

• i4 Apia UserJoe made an edit without a summary 

• 14 April UserJoe made an edit with summary: Fixed installation procedure step 5 


This wiki needs your help watching pages 


Few (if any) people are watching the pages below. They are related to other pages in your watchlist, and it'd real swell if 
you could help by watching some of them 


1. Watch: Manifest Dependencies 

2. Watch: Johnson Space Center 

3. Watch: Space Shuttle 

4 Watch: Control Moment Gyroscope 

5. Watch: EVA Skills 1 

6. Watch: PI ATAR&R 

7. Watch: Expedition 2 

8. Watch: 40S 

9. Watch: Expedition 5 

10. Watch: ISS OCAD 12345 


11 Watch: 38S 

12. Watch: Expedition 9 

13. Watch: 6" Wobble Socket 
14 Watch: Expedition 14 

15. Watch: FGB 
16 Watch: EVA Ratchet 

17. Watch: 34S 

18. Watch: EVA Hammer 

19. Watch: Expedition 17 

20. Watch: ISS Joint Airlock 




WATCH SUGGESTIONS: REAL CONVERSATION 

Disclaimer: the names are fake and the words are paraphrased, but the conversation is real. 

The other day Amy and Bob were heckling each other about who 

was watching more pages. 

• Amy: "I'm watching almost as many pages as our boss.J'm 
going to get more than her!" 

• Bob: "It just seems to randomly suggest pages to me, though. 
Why would I care about Rule #12345?" 

• Amy: "Well, that rule page is about your hardware." 

• Bob: "Oh, well maybe I do want to watch that! " 



HOW HAS WATCH SUGGESTIONS HELPED? 






BACK TO QUANTIFYING ACCOUNTABILITY 

1. Determine number of reviewers for each page 

2. Determine a way to rate contributors 

3. Combine (1) and (2) to identify weak points in our data 

4. Address weak points 



RATING REVIEWERS 

• Accuracy 

• Thoroughness 

• Impartiality 

• Variance across topics 

• Probability of correctness 

CORRECTNESS SCORE 



ENGAGEMENT SCORE 

Frequency and recency of editing 
Time between changes and reviews 
Use of page history and diff 
Time spent on each revision 
Quantity and age of pending reviews 



USER STATISTICS 


Watch Analytics: Users 

The state of the Wiki: 29014 watches of which 0.1% (42) are pending 
View: 

• Page watch statistics 

• User watch statistics 

• Wiki historical watch statistics 

• Visualization of all watches 


► ►! 

First page Previous page Next page Last page 


User 

# watches 

# pending watches 

% pending 

Max pending time 

Average pending time 

▲Engagement score 

Bob 

1108 

35 

3.1588 

54 days 

29 days 

0.533 

Larry 

8093 

3 

0.0371 

1 7 days 

12 days 

0.869 

Susan 

8085 

1 

0.0124 

1 day 

1 day 

0.980 

Sarah 

96 

2 

2.0833 

12 hours 

11 hours 

0.980 

George 

164 

1 

0.6098 

22 hours 

22 hours 

0.990 






PAGE WATCH QUALITY 


First page Previous page Next page Last page 


page 

# 

watches 

# reviewed 
watches 

% 

pending 

Max 

pending 

time 

Average 

pending 

time 

A 

Watch 

quality 

International Space Station (Watch) 

1 

1 

0.0000 



0.517 

Space Shuttle (Watch) 

1 

1 

0.0000 



0.635 

Apollo 13 (Watch) 

1 

1 

0.0000 



0.892 





PAGE WATCH QUALITY 


& James Talk Admin links Preferences My pending reviews (0) Contributions 



Contents [hide] 

1 Purpose 

2 Life Aboard 

2.1 Crew Operations 

2.2 Food 

2.3 Hygiene 

2.4 Repairs 







CULTURE CHANGES 



VOLUNTARY VS. ENTERPRISE 

• Contributing because you want to 

• Contributing because you are paid to 



GOALS 

Without continuous contributions and qualified 
reviews from everyone in your community, a wiki 
is just another dumpster of outdated and 
unreliable information. 



REVIEW INITIAL GOALS 

1. Determine number of reviewers for each page 

2. Determine a way to rate contributors 

3. Combine (1) and (2) to identify weak points in our data 

4. Address weak points 



REVIEW SHORT TERM GOALS 

1. Encourage users to do their reviews 

2. Eliminate singly-watched pages 



OUR GOALS FROM A DIFFERENT VIEW 

1. Increase number of watchers for each page 

2. Increase frequency of reviews performed on each page 

3. Increase quality of reviews performed on each page 



REFOCUS OUR GOALS 

Instead of focusing on adding content, our focus 
is now on the quality of our content 


Even Wikipedia is criticized for focusing on quantity over quality 

(Source) 



TARGET BEHAVIOR 

• Watch Pages 

• Edit often 

• Review promptly 

• Use page history and diff tool 

• Review carefully 

• Maintain low pending review count 



WHO ARE OUR USERS? 



EVA GROUP ANALYSIS 

• 743 viewers 

• 100 contributors 

• 47 active contributors 



RANGE OF RESPONSE 

Early adopters 

Those that began using after some help and motivation 
Viewers, but not contributors 
Viewers, contributors, but not reviewers 
Stubborn, refuse to even view the wiki 



MOTIVATION 

Extrinsic vs. Intrinsic 



INTRINSIC MOTIVATION 

• Competence 

• Autonomy 

• Relatedness 



USER PERSONALITY TYPES 

Based on an interview with Amy Jo Kim 

Competitors - Act on other users (challenge, showoff, compare) 
Self-Expressers - Act on content (create, build, design, customize) 
Explorers - Interact with content (view, collect, review, curate) 
Collaborators - Interacting with users (share, comment, help) 



ONLINE COMMUNITIES 

• In a zero-sum game, we are opponents: One wins, others lose 

• In a non-zero-sum game, we are partners: We all win or we all lose 



USER ACTIVITY LOOPS 



THE WIKI POLICE! 





EDUCATIONAL TOOLS 

• Welcome e-mail 

• Tutorial pages 

• Semantic Forms 

• Screencasts 

• Guided Tours 



MEETING MINUTES 


Create Meeting Minutes 


Remember... 

Sava often or um a teat aditor Raviatona turn troo Alio il thara >« anything you dont know how to do, aaa tha haip paga (or mo ra nfo 


Meeting Details 

Moating noma 
Moating data 
Start lima 
Notea takan by 

Meeting Fites 

Local Fite Kama or 
URL 

Alternate name 


(2*-*ou ten#, piatta) 



<*.♦ * i 


Add File 


Meeting Topics 


Add Topic 






SUBJECT MATTER EXPERT 



PENDING REVIEWS 





WHAT WE’VE BEEN MISSING 

Have we accounted for all four personality types? 

■ Competition 

■ Self-Expression 

■ Exploration 

■ Collaboration 

Have we provided for intrinsic motivation? 

■ Competence 

■ Autonomy 

■ Related ness 



WHAT WE PLAN TO ADD 



TARGET BEHAVIOR 

• Watch Pages 

• Edit often 

• Review promptly 

• Use page history and diff tool 

• Review carefully 

• Maintain low pending review count 
These behaviors are directly related to our new scores: 

• User Engagement Score 

• Page Scores 

These are still in development 



THANKS 



FUN 



GAMIFICATION FRAMEWORK 

1. Define objectives 

2. Delineate target behaviors 

3. Describe players 

4. Devise activity loops 

5. Don't forget the fun 

6. Deploy the appropriate tools 


Source: Professor Kevin Werbach, Wharton, University of Pennsylvania (Coursera) 



RESOURCES 

• Extension:WatchAnalytics 

• Extension:MeetingMinutes 

CONTACTS 

EnterpriseMediaWiki.org (github/@enterprisemw) 

• James Montalvo (github / @jamesmontalvo3) 

• Daren Welsh (github/ @darenwelsh) 

• Stephanie Johnston 



